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Hamada and Wu (1992) went beyond the traditional approach and proposed an analy- 
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Q . sis strategy to demonstrate that some interactions could be entertained and estimated 

beyond a few significant main effects. Their groundbreaking work stimulated much 
of the recent developments in design criterion creation, construction and analysis of 
nonregular designs. This paper reviews important developments in optimality criteria 
and comparison, including projection properties, generalized resolution, various general- 
ized minimum aberration criteria, optimality results, construction methods and analysis 
strategies for nonregular designs. 
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1 Introduction 

In many scientific investigations, the main interest is in the study of effects of many factors si- 
multaneously. Factorial designs, especially two-level or three-level factorial designs, are the most 
commonly used experimental plans for this type of investigation. A full factorial experiment allows 
all factorial effects to be estimated independently. However, it is often too costly to perform a full 
factorial experiment, so a fractional factorial design, which is a subset or fraction of a full factorial 
design, is preferred since it is cost-effective. 

Fractional factorial designs are classified into two broad types: regular designs and nonregular 
designs. Regular designs are constructed through defining relations among factors and are described 
in many textbooks such as Box, Hunter and Hunter (2005), Dean and Voss (1999), Montgomery 
(2005) and Wu and Hamada (2000). These designs have a simple aliasing structure in that any 
two effects are either orthogonal or fully aliased. The run sizes are always a power of 2, 3 or a 
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prime, and thus the "gaps" between possible run sizes are getting wider as the power increases. 
The concept of resolution (Box and Hunter 1961) and its refinement minimum aberration (Pries and 
Hunter 1980) play a pivotal role in the optimal choice of regular designs. There are many recent 
developments on minimum aberration designs; see Wu and Hamada (2000) and Mukerjee and Wu 
(2006) for further references. 

Nonregular designs such as Plackett-Burman designs and other orthogonal arrays are widely 
used in various screening experiments for their run size economy and flexibility (Wu and Hamada, 
2000). They fill the gaps between regular designs in terms of various run sizes and are flexible in 
accommodating various combinations of factors with different numbers of levels. Unlike regular 
designs, nonregular designs may exhibit a complex aliasing structure, that is, a large number 
of effects may be neither orthogonal nor fully aliased, which makes it difficult to interpret their 
significance. For this reason, nonregular designs were traditionally used to estimate factor main 
effects only but not their interactions. However, in many practical situations it is often questionable 
whether the interaction effects are negligible. 

Hamada and Wu (1992) went beyond the traditional approach and proposed an analysis strategy 
to demonstrate that some interactions could be entertained and estimated through their complex 
aliasing structure. They pointed out that ignoring interactions can lead to (i) important effects 
being missed, (ii) spurious effects being detected, and (iii) estimated effects having reversed signs 
resulting in incorrectly recommended factor levels. 

Much of the recent studies in nonregular designs were motivated from results in Hamada and Wu 
(1992). They included proposal of new optimality criteria, construction and analysis of nonregular 
designs. The primary aim of this paper is to review major developments in nonregular fractional 
factorial designs since 1992. 

Here is a brief history of the major developments in nonregular designs. Plackett and Bur- 
man (1946) gave a large collection of two-level and three-level designs for multi-factorial experi- 
ments. These designs are often referred to as the Plackett-Burman designs in the literature. Rao 
(1947) introduced the concept of orthogonal arrays, including Plackett-Burman designs as spe- 
cial cases. Cheng (1980) showed that orthogonal arrays are universally optimal for main effects 
model. Hamada and Wu (1992) successfully demonstrated that some interactions could be identi- 
fied beyond a few significant main effects for Plackett-Burman designs and other orthogonal arrays. 
Lin and Draper (1992) studied the geometrical projection properties of Plackett-Burman designs 
while Wang and Wu (1995) and Cheng (1995, 1998) studied the hidden projection properties of 
Plackett-Burman designs and other orthogonal arrays. The hidden projection properties provide 
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an explanation for the success of the analysis strategy due to Hamada and Wu (1992). Sun and Wu 
(1993) were the first to coin the term "nonregular designs" when studying statistical properties of 
Hadamard matrices of order 16. Deng and Tang (1999) and Tang and Deng (1999) introduced the 
concepts of generalized resolution and generalized minimum aberration for two-level nonregular de- 
signs. Xu and Wu (2001) proposed the generalized minimum aberration for mixed-level nonregular 
designs. Because of the popularity of minimum aberration, the research on nonregular designs has 
been largely focused on the construction and properties of generalized minimum aberration designs. 
Our reference list suggests that keen interest in nonregular designs began in 1999 and continues to 
this day as evident by the increasing number of scientific papers on nonregular designs in major 
statistical journals. 

Section 2 reviews the data analysis strategies for nonregular designs. Section 3 discusses the 
geometrical and hidden projection properties of the Plackett-Burman designs and other orthogonal 
arrays. Section 4 introduces the generalized resolution and generalized minimum aberration and 
their statistical justifications. Section 5 introduces the minimum moment aberration criterion, 
another popular criterion for nonregular designs. Section 6 considers uniformity and connections 
with various optimality criteria. Section 7 reviews construction methods and optimality results. 
Section 8 gives concluding remarks and future directions. 

2 Analysis Strategies 

We begin with a review of a breakthrough approach (Hamada and Wu 1992) by entertaining inter- 
actions in Plackett-Burman designs and other orthogonal arrays after identifying a few important 
main effects. Then we review another strategy proposed by Cheng and Wu (2001) for the dual 
purposes of factor screening and response surface exploration (or interaction detection) with quan- 
titative factors. 

The analysis strategy proposed by Hamada and Wu (1992) consists of three steps. 

Step 1. Entertain all the main effects and interactions that are orthogonal to the main effects. 
Use standard analysis methods such as ANOVA and half-normal plots to select significant 
effects. 

Step 2. Entertain the significant effects identified in the previous step and the two-factor interac- 
tions that consist of at least one significant effect. Identify significant effects using a forward 

selection regression procedure. 
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Step 3. Entertain the significant effects identified in the previous step and all the main effects. 
Identify significant effects using a forward selection regression procedure. 

Iterate between Steps 2 and 3 until the selected model stops changing. Note that the traditional 
analysis of Plackett-Burman or other nonregular designs ends at Step 1. 

Hamada and Wu (1992) based their analysis strategy on two empirical principles, effect sparsity 
and effect heredity (see Wu and Hamada 2000, Section 3.5). Effect sparsity implies that only few 
main effects and even fewer two-factor interactions are relatively important in a factorial experi- 
ment. Effect heredity means that in order for an interaction to be significant, at least one of its 
parent factors should be significant. Effect heredity excludes models that contain an interaction 
but none of its parent main effects, which lessens the problem of obtaining uninterpretable models. 
Hamada and Wu (1992) wrote that the strategy works well when both principles hold and the 
correlations between partially aliased effects are small to moderate. The effect sparsity suggests 
that only a few iterations will be required. 

Using this procedure, Hamada and Wu (1992) reanalyzed data from three real experiments, a 
cast fatigue experiment using a 12-run Plackett-Burman design with seven 2-level factors, a blood 
glucose experiment using an 18-run mixed-level orthogonal array with one 2-level and seven 3-level 
factors, and a heat exchange experiment using a 12-run Plackett-Burman design with ten 2-level 
factors. They demonstrated that the traditional main effects analysis was limited and the results 
were misleading. 

For illustration, consider the cast fatigue experiment conducted by Hunter, Hodi and Eager 
(1982) that used a 12-run Plackett-Burman design to study the effects of seven factors (A-G) on 
the fatigue life of weld repaired castings. Table 1 gives the data matrix and responses, where 
columns 8-11 are not used. The original analysis by Hunter, Hodi and Eager (1982) identified two 
significant factors F and D. The factor D had a much smaller effect with a p value around 0.2. 
The fitted model was 

y = 5.73 + 0.458F - 0.258D, (1) 

with a E? = 0.59. However, Hunter, Hodi and Eager (1982) noted a discrepancy between their 
fitted model ([T]) and previous work, namely, the sign of factor D was reversed. Applying the three- 
step analysis strategy, Hamada and Wu (1992) identified a significant two-factor interaction FG 
and obtained the following model 

y = 5.73 + 0.458F - 0.459FG. (2) 
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Table 1: Design Matrix and Responses, Cast Fatigue Experiment 

Factor Logged 



Run 


A 


B 


C 


D 


E 


F 


G 


8 


9 


10 


11 


Lifetime 


1 


+ 


+ 


- 


+ 


+ 


+ 


- 


- 


- 


+ 


- 


6.058 


2 


+ 


- 


+ 


+ 


+ 


- 


- 


- 


+ 


- 


+ 


4.733 


3 


- 


+ 


+ 


+ 


- 


- 


- 


+ 


- 


+ 


+ 


4.625 


4 


+ 


+ 


+ 


- 


- 


- 


+ 


- 


+ 


+ 


- 


5.899 


5 


+ 


+ 


- 


- 


- 


+ 


- 


+ 


+ 


- 


+ 


7.000 


6 


+ 


- 


- 


- 


+ 


- 


+ 


+ 


- 


+ 


+ 


5.752 


7 








+ 




+ 


+ 




+ 


+ 


+ 


5.682 


8 






+ 




+ 


+ 




+ 


+ 


+ 




6.607 


9 




+ 




+ 


+ 




+ 


+ 


+ 






5.818 


10 


+ 




+ 


+ 




+ 


+ 


+ 








5.917 


11 




+ 


+ 




+ 


+ 


+ 








+ 


5.863 


12 
























4.809 



This model has B? = 0.89, which is a significant improvement over model ([T]) in terms of goodness 
of fit. The identification of FG was not only consistent with the engineering knowledge reported 
in Hunter, Hodi and Eager (1982) but also provided a sound explanation on the discrepancy of the 
sign of factor D. The coefficient of D in ([T]) actually estimates D + ^FG and therefore the sign of D 
in ([T]) could be negative even if D had a small positive effect. This experiment was later reanalyzed 
with other methods by several authors, including Box and Meyer (1993), Chipman, Hamada and 
Wu (1997), Westfall, Young and Lin (1998), Yuan, Joseph and Lin (2007), and Phoa, Pan and Xu 
(2007). 

Hadama and Wu (1992) discussed limitations of their analysis strategy and provided solutions. 
Wu and Hamada (2000, chap. 8) further suggested some extensions such as the use of all subset 
variable selection if possible. 

For quantitative factors with more than two levels, Cheng and Wu (2001) proposed the following 
two-stage analysis strategy to achieve the dual objectives of factor screening and response surface 
exploration (or interaction detection) using a single design. This two-stage analysis strategy is also 
the two key aspects in standard response surface methodology. 
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Stage 1. Perform factor screening and identify important factors. 

Stage 2. Fit a second-order model for the factors identified in stage 1. 
For m quantitative factors, denoted by xi, . . ., Xm, the second-order model is 

mm m 

y = Po + ^ PiXi + Piixl + ^ PijXiXj + e, 

j=l i=l l=i<j 

where Po, Pi, f^u, are unknown parameters and e is the error term. 

Cheng and Wu (2001) proposed that the two-stage analysis be broken down into three parts: 
screening analysis in stage 1, projection that links stages 1 and 2, and response surface exploration 
in stage 2. Various screening analyses can be utilized in stage 1, such as the conventional ANOVA or 
half-normal plots on the main effects. Their analysis strategy again assumes that effect sparsity and 
effect heredity principles hold. They reanalyzed a PVC insulation experiment reported by Taguchi 
(1987) that used a regular 27-run design with nine 3-level factors. They identified a significant 
linear-by-linear interaction effect which was missed by Taguchi. 

For illustration, consider an experiment reported by King and Allen (1987) that used an 18-run 
orthogonal array to study the effects of one two-level factor {A) and seven three-level factors {B-H) 
on radio frequency chokes. Each run had two replicates and Table 2 gives the design matrix and 
the responses. Xu, Cheng and Wu (2004) performed data analysis following the two-stage analysis 
strategy. At the first stage, they fitted an ANOVA model for main effects and found that four 
factors B, E, G, and H were significant at the usual 5% level. At the second stage, they fitted a 
second-order model among the four active factors and obtained the following nine-effect model: 

y = 105.1 + 2MB - A.ObE - 7.75G + 2MH - 2.85E^ + 1.39BE - 3.3QEG - lAlEH + 1.86GH, 

where the levels 0, 1, 2 were coded as —1,0, 1, respectively. The model has = 0.96, indicating 
a good fit. It is worthwhile to point out that the 18-run design does not have enough degrees 
of freedom to estimate all six two-factor interactions among four factors (since each two-factor 
interaction has four degrees of freedom). 

More sophisticated analysis strategics have been proposed for experiments with complex alias- 
ing. Box and Meyer (1993) proposed a Bayesian method for finding the active factors in screening 
experiments. Chipman, Hamada and Wu (1997) proposed a Bayesian approach that employs a 
Gibbs sampler to perform an efficient stochastic search of the model space. Many other recent 
variable selection methods can also be used for analyzing nonregular designs. For example. Yuan, 
Joseph and Lin (2007) suggested an extension of the general-purpose LARS (least angle regression), 
first proposed by Efron, Hastie, Johnstone and Tibshirani (2004). 
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Table 2: Design Matrix and Responses, Radio Frequency Chokes Experiment 



Run 


A 


B 


C 


D 


E 




G 




Responses 


1 


























106.20 


107.70 


2 








1 


1 


1 


1 


1 


1 


104.20 


102.35 


3 








2 


2 


2 


2 


2 


2 


85.90 


85.90 


4 





1 








1 


1 


2 


2 


101.15 


104.96 


5 





1 


1 


1 


2 


2 








109.92 


110.47 


6 





1 


2 


2 








1 


1 


108.91 


108.91 


7 





2 





1 





2 


1 


2 


109.76 


112.66 


8 





2 


1 


2 


1 





2 





97.20 


94.51 


9 





2 


2 





2 


1 





1 


112.77 


113.03 


10 


1 








2 


2 


1 


1 





93.15 


92.83 


11 


1 





1 








2 


2 


1 


97.25 


100.6 


12 


1 





2 


1 


1 








2 


109.51 


113.28 


13 


1 


1 





1 


2 





2 


1 


85.63 


86.91 


14 


1 


1 


1 


2 





1 





2 


113.17 


113.45 


15 


1 


1 


2 





1 


2 


1 





104.85 


98.87 


16 


1 


2 





2 


1 


2 





1 


113.14 


113.78 


17 


1 


2 


1 





2 





1 


2 


103.19 


106.46 


18 


1 


2 


2 


1 





1 


2 





95.70 


97.93 



7 



3 Projection Properties of Plackett-Burman Designs and Other 
Orthogonal Arrays 



We first review the concept of orthogonal arrays due to Rao (1947). An orthogonal array of N 
runs, m factors, s levels and strength t, denoted by OA{N, s'^,t), is an N xm matrix in which each 
column has s symbols or levels and for any t columns all possible s* combinations of symbols appear 
equally often in the matrix. Rao (1973) generalized the definition to the asymmetrical case where 
an orthogonal array is allowed to have variable numbers of symbols, i.e., mixed levels. For example, 
the 12-run Plackett-Burman design in Table 1 is an OA{12, 2^^, 2) and the 18-run design in Table 
2 is an OA(18, 2^3'', 2). Hedayat, Sloane and Stufken (1999) gave a comprehensive description on 
various aspects of orthogonal arrays. 

Plackett-Burman designs arc saturated orthogonal arrays of strength two because all degrees of 
freedom arc utilized to estimate main effects. Orthogonal arrays of strength two allow all the main 
effects to be estimated independently and they are universally optimal for the main effects model 
(Cheng 1980). A necessary condition for the existence of an OA{N, s™, 2) is that N — 1 > m{s — 1). 
A design is called saturated ii N — 1 = m{s — 1) and supersaturated if — 1 < m{s — 1). In 
the literature, orthogonal arrays of strength two are often called orthogonal designs or orthogonal 
arrays without mentioning the strength explicitly. 

Orthogonal arrays include both regular and nonregular designs. For regular designs, the con- 
cepts of strength and resolution are equivalent because a regular design of resolution R is an 
orthogonal array of strength t = R — 1. For a regular design of resolution R, the projection onto 
any R factors must be either a full factorial or copies of a half-replicate of a full factorial. The 
projection for nonregular designs is more complicated. 

Plackett-Burman designs are of strength two so that the projection onto any two factors is a 
full factorial. Lin and Draper (1992) studied the geometrical projection properties of the Plackett- 
Burman designs onto three or more factors. Their computer searches found all the projections of 
12-, 16-, 20-, 24-, 28-, 32- and 36-run Plackett-Burman designs onto three factors. They found 
that these projections must have at least a copy of the full 2^ factorial or at least a copy of a 
2^~^ replicate or both. In particular, any projection onto three factors must contain a copy of a 
full factorial except for the 16- and 32-run Plackett-Burman designs, which are regular designs. 
The important statistical implication of this finding is that if only at most three factors are truly 
important, then after identifying the active factors, all factorial effects among these active factors 
are estimable, regardless which three factors are important. 
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Box and Tyssedal (1996) defined a design to be of projectivity p if the projection onto every 
subset of p factors contains a full factorial design, possibly with some points replicated. It follows 
from these definitions that an orthogonal array of strength t is of projectivity t. Cheng (1995) 
showed that, as long as the run size N is not a multiple of 2*+^, an OA{N, 2"^,t) with m > t + 2 
has projectivity t + 1, even though the strength is only t. 

The 12-run Plackett-Burman design given in Table 1 is of projectivity three but not of projectiv- 
ity four. Wang and Wu (1995) found that its projection onto any four factors has the property that 
all the main effects and two-factor interactions can be estimated if the higher-order interactions are 
negligible. They referred this estimability of interactions without relying on geometric projection 
to as having a hidden projection property. 

More generally, Wang and Wu (1995) defined a design as having a hidden projection property if 
it allows some or all interactions to be estimated even when the projected design does not have the 
right resolution or other geometrical/combinatorial design property for the same interactions to be 
estimated. For the Plackctt-Burman designs their hidden projection property is a result of complex 
aliasing between the interactions and the main effects. For example, in the 12-run Plackett-Burman 
design given in Table 1, any two-factor interaction, say AB, is orthogonal to the main effects A 
and B, and partially aliased with all other main effects with correlation 1/3 or —1/3. Because no 
two-factor interaction is fully aliased with any main effects, it is possible to estimate four main 
effects and all six two-factor interactions among them together. 

The general results on hidden projection properties were obtained by Cheng (1995, 1998) and 
Bulutoglu and Cheng (2003). Cheng (1995) showed that as long as the run size N of an OA{N, 2"*, 2) 
is not a multiple of 8, its projection onto any four factors allows the estimation of all the main 
effects and two-factor interactions when the higher-order interactions are negligible. Bulutoglu 
and Cheng (2003) showed that the same hidden projection property also holds for Paley designs 
[constructed by a method due to Paley (1933)] of sizes greater than 8, even when their run sizes are 
multiples of 8. A key result is that such designs do not have defining words of length three or four. 
Cheng (1998) further showed that as long as the run size N of an OA{N, 2™", 3) is not a multiple of 
16, its projection onto any five factors allows the estimation of all the main effects and two-factor 
interactions. Cheng (2006) gave a nice review of projection properties of factorial designs and their 
role in factor screening. 

A few papers studied projection properties of designs with more than two levels. Wang and 
Wu (1995) studied the hidden projections onto 3 and 4 factors of the popular 0^(18,3^,2) given 
in Table 2 (columns B-H). Cheng and Wu (2001) further studied the projection properties of this 
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OA{18, 3^, 2) and an OA{36, 3^^, 2) in terms of their two-stage analysis strategy. They constructed 
a nonregular 0^4(27,3^,2) that allows the second-order model to be estimated in all four-factor 
projections. In contrast, any regular 27-run design with eight 3-level factors does not have this 
four-factor projection property. They concluded that three-level nonregular designs have better 
projection properties and are more useful than regular designs for the dual purposes of factor 
screening and response surface exploration. Xu, Cheng and Wu (2004) further explored the projec- 
tion properties of 18-run and 27-run orthogonal arrays and constructed a nonregular OA{27, 3^'^, 2) 
that allows the second-order model to be estimated in all of the five-factor projections. Tsai, 
Gilmour and Mead (2000, 2004), Evangelaras, Koukouvinos, Dean and Dingus (2005) and Evan- 
gelaras, Koukouvinos and Lappas (2007, 2008) also studied projection properties of three-level 
orthogonal arrays. Dcy (2005) studied projectivity properties of asymmetrical orthogonal arrays 
with all except one factors having two levels. 

4 Generalized Resolution and Generalized Minimum Aberration 

Prior to 1999, an outstanding problem was how to assess, compare and rank nonregular designs in 
a systematic fashion. Deng and Tang (1999) and Tang and Deng (1999) were the first to propose 
generalized resolution and generalized minimum aberration criteria for 2-level nonregular designs, 
which are natural generalizations of the traditional concepts of resolution and minimum aberration 
for regular designs. 

To define these two important concepts, generalized resolution and generalized minimum aber- 
ration, Deng and Tang (1999) and Tang and Deng (1999) introduced the important notion of 
J-characteristics and is defined as follows. Given a two- level N x m design D = (dij), for 
s = {ci, . . . , Cfe}, a subset of k columns of D, define 



where Cij is the ith component of column Cj. The quantity jkis)/N can be viewed as an extension 
of correlation. For illustration, consider the 12-run Plackett-Burman design given in Table 1. For 
s = {A,B}, j2is) = since A and B are orthogonal. For s = {A, B,C}, j3(s)/A^ = —1/3 
is the correlation between main effect A and two-factor interaction BC. For s = {A, B,C, D}, 
ji{s)/N = —1/3 is the correlation between two two-factor interactions, say AB and CD. The 
quantity Pk{s) = Jk{s)/N is called the normalized J-characteristics by Tang and Deng (1999) or 
aliasing index by Cheng, Li and Ye (2004) and Phoa and Xu (2008) because < pfe(s) < 1. It is 



N 




(3) 



i=l 
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not difficult to see that if D is a two-level regular design then pfe(s) = or 1 for all s. Ye (2004) 
showed that the reserve is also true. Therefore, for a nonregular design, there always exist some s 
such that < /9fe(s) < 1. 



For the 12-run design in Table 1, r = 3, (5 = 2/3 and the generalized resolution is i? = 3.67. It is 
easy to see that for an OA{N,2"^ ,t), jfe(s) = for any k < t and therefore r < R < r + 1 where 
r = t + 1. If(5>0, a subset s of D with r columns contains at least N6/2^' copies of a full 2^ 
factorial and therefore the projectivity of D is at least r (Deng and Tang (1999)). For a regular 
design, 6 = and the projectivity is exactly r — 1. 

Two regular designs of the same resolution can be distinguished using the minimum aberration 
criterion, and the same idea can be applied to nonregular designs using the minimum G -aberration 
criterion (Deng and Tang (1999)). Roughly speaking, the minimum G-aberration criterion always 
chooses a design with the smallest confounding frequency among designs with maximum gener- 
alized resolution. Formally, the minimum G-aberration criterion is to sequentially minimize the 
components in the confounding frequency vector 



where fkj denotes the frequency of /c-column combinations s with Jk{s) = N + 1 ~ j . 

Minimum G-aberration is very stringent and it attempts to control J-characteristics in a very 
strict manner. Tang and Deng (1999) proposed a relaxed version of minimum G-aberration and 
called it the minimum G2-aberration criterion. Let 



The vector {Ai{D), . . . ,Afn{D)) is called the generalized wordlength pattern, because for a regular 
design D, Ak{D) is the number of words of length k in the defining contrast subgroup of D. 
The minimum G2-aberration criterion (Tang and Deng (1999)) is to sequentially minimize the 
generalized wordlength pattern Ai{D), A2{D), . . . , AmiD). 

For regular designs both minimum G-aberration and minimum G2-aberration criteria reduce to 
the traditional minimum aberration criterion. However, these two criteria can result in selecting 




CFV(D) = [(/n, . . . , /liv); (/2i, . . . , /2jv); . . . ; 



(/ml) ■ ■ ■ ) fmN)], 




(5) 



\s\=k 
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different nonregular designs. We note that minimum G-aberration nonregular designs always have 
maximum generalized resolution whereas minimum G2-aberration nonregular designs may not. This 
is in contrast to regular case where minimum aberration regular designs always have maximum 
resolution among all regular designs. 

Tang and Deng (1999) also defined minimum Gg-aberration for any e > by replacing J|(s) 
with J|(s) in ([5]). However, only the minimum G2-aberration criterion is popular due to various 
statistical justifications and theoretical results. 

Xu and Wu (2001) proposed the generalized minimum aberration criterion for comparing asym- 
metrical (or mixed-level) designs. The generalized minimum aberration criterion was motivated 
from ANOVA models and includes the minimum G2-aberration criterion as a special case. By 
exploring an important connection between design theory and coding theory, Xu and Wu (2001) 
showed that the generalized wordlength pattern defined in ([5]) are linear combinations of the distri- 
bution of pairwise distance between the rows. This observation plays a pivotal role in the subsequent 
theoretical development of nonregular designs. 

Ma and Fang (2001) independently extended the minimum G2-aberration criterion for designs 
with more than two levels. They named their criterion as the minimum generalized aberration 
criterion, which is a special case of the generalized minimum aberration criterion proposed by Xu 
and Wu (2001). 

Ye (2003) redefined the generalized wordlength pattern and generalized minimum aberration 
for two- level designs using indicator functions. Cheng and Ye (2004) defined generalized resolution 
and generalized minimum aberration criterion for quantitative factors. The generalized minimum 
aberration criterion proposed by Xu and Wu (2001) is independent of the choice of treatment 
contrasts and thus model-free whereas the generalized minimum aberration criterion by Cheng and 
Ye (2004) depends on the specific model. 

4.1 Statistical Justifications 

Deng and Tang (1999) provided a statistical justification for the generalized resolution by showing 
that designs with maximum generalized resolution minimize the contamination of nonnegligible 
two-factor interactions on the estimation of main effects. Tang and Deng (1999) provided a similar 
statistical justification for minimum G2-aberration designs. In a further extension, Xu and Wu 
(2001) gave a statistical justification for generalized minimum aberration designs with mixed levels. 
A common situation that arises in practice is that the main effects are of primary interest but 
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there are uninteresting yet non-negligible interactions that we know will affect the main effects 
estimates. To fix ideas, consider a two-level N x m design D = (dij) with columns denoted by 
di, . . . ,dm and generalized resolution between 3 and 4. Suppose that one fits a main effects model 

m 

yi = Po + ^Pjdij + ei, (6) 
i=i 

but the true model is 

m m 

yi = Pq + ^ Pjdij + Pkidikdii + ei. (7) 

j=l k<l 

The least squares estimator Pj of Pj from the working model ([6]), under the true model ([7]), has 
expectation given by 

m 

EiPj) = pj + iv-i 5]i3(d,-, 4, di)pki 

k<l 

for j = l,...,m, where j^{dj,dk,di) is defined in ([3]). There are many ways to minimize the 
biases in estimating main effects due to the presence of the interaction effects. A conservative 
approach is minimizing the maximum bias, maxj ^k<l Jsidj , dk , di) . This is equivalent to max- 
imizing the generalized resolution as defined in Therefore, designs with maximum gener- 
alized resolution minimize the maximum bias of nonnegligible interactions on the estimation of 
the main effects. A more aggressive approach is minimizing the sum of squared coefficients 
Y.f=iY.T<i[h{dj,dk,di)/N]^ = SAsiD), where AsiD) is defined in i^. Hence minimum G2- 
aberration designs minimize the overall contamination of nonnegligible interactions on the esti- 
mation of the main effects. 

For regular designs, Cheng, Steinberg and Sun (1999) justified the minimum aberration crite- 
rion by showing that it is a good surrogate for some model-robustness criteria. Following their 
approach, Cheng, Deng and Tang (2002) considered the situation where (i) the main effects are 
of primary interest and their estimates are required and (ii) the experimenter would like to have 
as much information about two-factor interactions as possible, under the assumption that higher- 
order interactions are negligible. Without knowing which two-factor interactions are significant, 
they considered the set of models containing all of the main effects and / two-factor interactions for 
/ = 1, 2, 3, . . .. Let Ef be the number of estimable models and Df be the average of Z)-efficiencies. 
Cheng, Deng and Tang (2002) showed that the minimum G2-aberration designs tend to have large 
Ef and Df values, especially for small /; therefore, the minimum G2-aberration criterion provides 
a good surrogate for the traditional model-dependent efficiency criteria. Ai, Li and Zhang (2005) 
and Mandal and Mukerjee (2005) extended their approach to mixed-level designs. 
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5 Minimum Moment Aberration 



Based on coding theory, Xu (2003) proposed the minimum moment aberration criterion for assessing 
nonregular designs. For an x m design D with s levels and a positive integer t, define the rth 
power moment to be 

Km = [N{N-l)/2]-' Y: MD)f, (8) 

l<i<j<N 

where Sij{D) is the number of coincidences between the ith and jth rows. For two row vectors 
(xi, . . . , Xm) and (yi, . . . , Um), the number of coincidences is the number of i's such that Xj = yi. 
Note that Tn — 6ij{D) is known as the Hamming distance between the ith. and jth rows in coding 
theory. 

The power moments measure the similarity among runs (i.e., rows). The first and second power 
moments measure the average and variance of the similarity among runs. Minimizing the power 
moments makes runs to be as dissimilar as possible. Therefore, good designs should have small 
power moments. This leads to the minimum moment aberration criterion (Xu (2003)) that is to 
sequentially minimize the power moments Ki{D), K2{D), . . . , Kjn{D). 

We note that the computation of the power moments involves the number of coincidences 
between rows. By applying generalized Mac Williams identities and Pless power moment identities, 
two fundamental results in coding theory (see, e.g., Mac Williams and Sloane 1977, chap. 5), Xu 
(2003) showed that the power moments Kt defined in ^ are linear combinations of the wordlength 
patterns Ai, . . . ,At in Specifically, 

KtiD) = ctAtiD) + Q_i^_i(Z)) + . . . + ciAiiD) + cq, 

where Cj are constants depending on i, N, m, s only and the leading coefficient q is positive. It is not 
difficult to see now that sequentially minimizing Ki{D), . . . ,Km{D) is equivalent to sequentially 
minimizing Ai{D), . . . ,Afn{D)- Therefore, the minimum moment aberration is equivalent to the 
generalized minimum aberration. 

The equivalence of the minimum moment aberration and the generalized minimum aberration 
is very important. On the one hand, it not only provides a geometrical justification for the gener- 
alized minimum aberration, but also provides a statistical justification for the minimum moment 
aberration. On the other hand, it provides a useful tool for efficient computation and theoretical 
development. For an x m design with two levels, the complexity of computing the generalized 
wordlength pattern according to the definition ([5]) is 0{N2'^) whereas the complexity of computing 
m power moments is O(A^^m^). The saving in computation is tremendous when the number of 
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factors m is large. This observation led to successful algorithmic constructions of mixed-level or- 
thogonal arrays (Xu (2002)), a catalog of 3- level regular designs (Xu (2005b)), and blocked regular 
designs with minimum aberration (Xu and Lau (2006)). As a theoretical tool, Xu (2003) developed 
a unified theory for nonregular and supersaturated designs. Xu and Lau (2006) and Xu (2006) 
further used the concept of minimum moment aberration to develop a theory for blocked regular 
designs and constructed minimum aberration blocked regular designs with 32, 64 and 81 runs. 

To mimic the minimum G-aberration criterion (Deng and Tang (1999)), Xu and Deng (2005) 
applied the minimum moment aberration criterion to projection designs and proposed the moment 
aberration projection to rank and classify general nonregular designs. It was a surprise that the 
minimum G-aberration criterion and the moment aberration projection criterion are not equivalent 
for two-level designs. Xu and Deng (2005) provided examples to show that the latter is more pow- 
erful in classifying and ranking nonregular designs than the former. They also provided examples 
to illustrate that the moment aberration projection criterion is supported by other design criteria. 
The concept of moment projection turns out to be very useful in the algorithmic construction of 
regular designs; see Xu (2005b, 2007). 

For mixed-level designs, Xu (2003) suggested to weight each column according to its level, called 
natural weights, and replace 5ij{D) in ([8]) with the number of weighted coincidences. Xu (2003) 
showed that the minimum moment aberration with natural weights is weakly equivalent to the 
generalized minimum aberration for mixed-level designs. 

6 Uniformity and Connection Among Various Criteria 

Uniformity or space filling is a desirable design property for computer experiments (Fang, Li and 
Sudjianto (2006)). Various uniformity measures are used to assess the space filling property for 
the so-called uniform design (Fang and Wang (1994), Fang, Lin, Winker and Zhang (2000)). Fang 
and Mukerjee (2000) found a connection between aberration and uniformity for 2-level regular 
designs. This connection was extended by Ma and Fang (2001) for general two-level designs. The 
basic result states that for a two-level N x m design D, regular or nonregular, the centered L2- 
discrepancy {CL2), a uniformity measure introduced by Hickernell (1998), can be expressed in 
terms of its generalized wordlength pattern ^^(D) as follows: 

™>-(i)"'-(i)'"^(r{-j^f 
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Since the coefHcient of Ak{D) decreases exponentially with k, one can anticipate that designs 
with small Ak{D) for small values of k should have small {CL2(-D)}^; in other words, minimum 
G2-aberration designs tend to be uniform over the design region. Ma and Fang (2001) also gave 
analytic formulas that link the generalized wordlength pattern with other uniformity measures for 
two- and three-level designs. 

Tang (2001) showed that minimum G2-aberration designs have good low-dimensional projection 
properties. Ai and Zhang (2004a) extended this result to mixed-level designs and showed that 
generalized minimum aberration designs have good low-dimensional projection properties. 

There is much more work on the connection among aberration, uniformity and projection. 
Hickcrncll and Liu (2002) showed that generalized minimum aberration designs and minimum dis- 
crepancy designs arc equivalent in a certain limit. Qin and Fang (2004), Ai, Li and Zhang (2005), 
Fang and Qin (2005), Liu, Fang and Hickcrncll (2006), Qin and Ai (2007), and Qin, Zou and 
Chatterjee (2008) discussed the connections among different criteria for symmetrical and asym- 
metrical fractional factorial designs, including generalized minimum aberration, minimum moment 
aberration, and various uniformity measures. 

7 Construction and Optimality Results 

An important and challenging issue is the construction of good nonrcgular designs. There are two 
simple reasons: (i) nonregular designs do not have a unified mathematical description; (ii) there 
are much more nonregular designs than regular designs. Since 1999, a main stream of researches 
focused on searching or constructing nonregular designs with good properties in terms of the mini- 
mum G2-aberration and generalized minimum aberration criteria. This section reviews algorithmic 
constructions and optimality results. The last subsection reviews a simple yet powerful construction 
method via quaternary codes. 

7.1 Algorithmic constructions 

Two-level nonrcgular designs are often constructed from Hadamard matrices. A Hadamard matrix 
of order is an x matrix with the elements ±1 whose columns (and rows) arc orthogonal 
to each other. From a Hadamard matrix of order N, one obtains a saturated two-level orthog- 
onal array with N runs and A — 1 columns, which is a nonregular design if A is not a power 
of 2. Neil Sloane of AT&T Shannon Labs maintains a large collection of Hadamard matrices 
at his website http: //www. research. att. com/ ^^nj as/, which includes all Hadamard matrices of 
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orders N up through 28, and at least one of every order N up through 256. Sloane also main- 
tains a library of orthogonal arrays as a companion to the book by Hedayat, Sloane and Stufken 
(1999). SAS maintains a library of orthogonal arrays (of strength two) up through 144 runs at 
http: //support . sas . com/techsup/technote/ts723 .html. SAS also provides a set of free macros 
for constructing over 117,000 orthogonal arrays up through 513 runs, which are documented in the 
free Web book by Kuhfeld (2005). 

A simple strategy for constructing generalized minimum aberration designs is searching over all 
possible projection designs from existing Hadamard matrices or orthogonal arrays. Deng and Tang 
(2002) presented a catalog of generalized minimum aberration designs by searching over Hadamard 
matrices of order 16, 20, and 24. However, limiting to Hadamard matrices may miss the optimal 
design in some cases; therefore, Li, Tang and Deng (2004) searched generalized minimum aberration 
designs from and outside Hadamard matrices with 20, 24, 28, 32 and 36 runs. They found that 
the best 20 x 6 and 20 x 7 designs according to minimum G-aberration cannot be obtained from 
Hadamard matrices. Similarly, Xu and Deng (2005) considered the construction of optimal designs 
under the moment aberration projection criterion. Besides searching over all Hadamard matrices 
of order 16 and 20, they searched over all projection designs from 68 saturated OA{27, 3^^, 2) from 
Lam and Tonchev (1996). They also observed that not all 20-run and 27-run moment aberration 
projection designs can be embedded into Hadamard matrices or saturated orthogonal arrays. 

Sun, Li and Ye (2002) proposed an algorithm for sequentially constructing non-isomorphic 
orthogonal designs. Two designs are said to be isomorphic or equivalent if one design can be 
obtained from the other by row permutations, column permutations, or relabeling of levels. An 
essential element of their algorithm is using minimal column base to reduce the computations for 
determining isomorphism between any two designs. By using this algorithm, they obtained the 
complete catalogs of two-level orthogonal designs for 12, 16, and 20 runs. Their results suggest 
that there is only one unique 12 x m design for m = 4 and 7 < m < 11 and that there are two 
non-isomorphic 12 x m design for m = 5 and 6. All these designs can be found as projection designs 
of the 12-run Plackett-Burman design given in Table 1. They found that there are five 16 x 15 
orthogonal designs, which are equivalent to the five non-isomorphic Hadamard matrices of order 
16 by Hall (1961). An important result is that all 16-run orthogonal designs are projections of one 
of the five 16-run Hadamard matrices. They found that there are three 20 x 19 orthogonal designs, 
which are equivalent to the three non-isomorphic Hadamard matrices of order 20 by Hall (1965). 
From their complete catalog, they obtained generalized minimum aberration designs. They found 
that most of the generalized minimum aberration designs are projections of the 20-run Hadamard 
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matrices and thus agrees with the designs reported in Deng and Tang (2002). However, they found 
the generahzed minimum aberration designs for m = 6 and m = 7 are not projections of the 
Hadamard matrices. This agrees with the results from Li, Tang and Deng (2004) and Xu and Deng 
(2005). The complete catalogs of 12, 16 and 20 runs were later used by Li, Lin and Ye (2003) in 
the choice of optimal foldover plans, by Cheng, Li and Ye (2004) in the construction of blocked 
nonregular designs, by Loeppky, Bingham and Sitter (2006) for constructing nonregular robust 
parameter designs, and by Li (2006) for constructing screening designs for model selection. 

Xu, Cheng and Wu (2004) considered the design issues related to the dual objectives of factor 
screening and interaction detection for quantitative factors. They proposed a set of optimality 
criteria to assess the performance of designs and a three-step approach to searching for optimal 
designs. They not only searched over all projection designs from the commonly used 0^(18,3^,2) 
given by columns i? to in Table 2 and 68 saturated OA{27, 3^^, 2) from Lam and Tonchev (1996), 
but also used an algorithm due to Xu (2002) to construct new designs directly. They presented 
many efficient and practically useful three-level nonregular designs with 18 and 27 runs for the dual 
objectives. Evangelaras, Koukouvinos and Lappas (2007) completely enumerated all nonisomorphic 
orthogonal arrays with 18 runs and 3 levels. Their results suggest that there are 4, 12, 10, 8, and 
3 nonisomorphic OA(18, 3"*,2) for m = 3, 4, 5, 6, and 7, respectively. Evangelaras, Koukouvinos 
and Lappas (2008) further completely enumerated all nonisomorphic 0^(27,3"*, 2) for m = 3-13 
and identified 129 nonisomorphic saturated 0^(27,3^^,2). 

Loeppky, Sitter and Tang (2007) proposed to rank two-level orthogonal designs based on the 
number of estimable models containing a subset of main effects and their associated two-factor 
interactions. They argued that by ranking designs in this way, the experimenter can directly assess 
the usefulness of the experimental plan for the purpose in mind. They presented catalogs of useful 
designs with 16, 20, 24, and 28 runs. 

All these algorithmic constructions are limited to small run sizes (< 32) due to the existence 
of a large number of designs and the difhculty of determining whether two designs are isomorphic 
or equivalent. Katsaounisa and Dean (2008) gave a survey and evaluation of methods for deter- 
mination of equivalence of factorial designs. Fang, Zhang and Li (2007) proposed an optimization 
algorithm for constructing generalized minimum aberration designs. It is not clear how effective 
their algorithm is for constructing large designs. Bulutoglu and Margot (2008) recently completely 
classified some orthogonal arrays of strength 3 up to 56 runs and of strength 4 up to 144 runs. 
However, these arrays have a small number of factors (< 11). 
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7.2 Optimality and Theoretical Results 

A powerful tool in the study of regular designs is the complementary design technique. Every 
regular design can be determined by a unique complementary design. It is convenient to study the 
complementary design when it is small. Tang and Deng (1999) developed a complementary design 
theory for minimum G2-aberration nonregular designs and Xu and Wu (2001) further developed 
a theory for generalized minimum aberration designs. The theory was extended by Ai and Zhang 
(2004b) for blocked nonregular designs and by Ai and He (2006) for nonregular designs with multiple 
groups of factors, including robust parameter designs. However, unlike in the regular case, a 
nonregular design can have none, one or more than one complementary designs; therefore, the 
complementary design theory for nonregular designs is less useful than that for regular designs. 

Xu (2003) gave several sufficient conditions for a design to have minimum moment aberration 
and generalized minimum aberration among all possible designs. One sufficient condition is that for 
an orthogonal array of strength t its projection onto any t + 1 columns does not have repeated runs. 
For example, consider the OA(18, 3^,2) given by columns C to in Table 2. It is easy to verify 
that its projection onto any three columns does not have repeated runs. Thus, this design (and any 
of its projections) has minimum moment aberration and generalized minimum aberration among all 
possible designs. Another sufficient condition is that the numbers of coincidences between distinct 
rows are constant or differ by at most one. In other words, a design is optimal under the minimum 
moment aberration and generalized minimum aberration criteria if its design points are eqiially or 
nearly equally spaced over the design region. As an example, the Oyl(12, 2^^, 2) given in Table 1 
is optimal because the number of coincidences between any two distinct rows is 5. Generalizing 
this, Zhang, Fang, Li and Sudjianto (2005) proposed a majorization framework and showed that 
orthogonality, aberration and uniformity criteria can be unified by properly choosing combinatorial 
and exponential kernels. 

Tang and Deng (2003) presented construction methods that yields maximum generalized reso- 
lution designs for 3, 4 and 5 factors and any run size that is a multiple of 4. Butler (2003, 2004) 
presented a number of construction results that allow minimum G2-aberration designs to be found 
for many of the cases with N =16, 24, 32, 48, 64 and 96 runs. Butler (2005) further developed 
theoretical results and presented methods that allow generalized minimum aberration designs to 
be constructed for more than two levels. A key tool used by Butler (2003, 2004, 2005) is some 
identities that link the generalized wordlength patterns with moments of the inner products or 
Hamming distances between the rows. These identities can be derived easily from the generalized 
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Pless power moment identities developed by Xu (2003). 

Xu (2005a) constructed several nonregular designs with 32, 64, 128, and 256 runs and 7-16 
factors from the Nordstrom and Robinson code, a well-known nonlinear code in coding theory. 
These designs are better than regular designs of the same size in terms of resolution, aberration and 
projectivity. By using linear programming he showed that 13 nonregular designs have minimum 
G2-aberration among all possible designs and seven orthogonal arrays have unique generalized 
wordlength patterns. 

Tang (2006) studied the existence and construction of orthogonal arrays that are robust to 
nonnegligible two-factor interactions. Butler (2007) showed that foldover designs are the only 
(regular or nonregular) two- level factorial designs of resolution IV or more for N runs and A^/3 < 
m < N/2 factors. Yang and Butler (2007) studied two-level nonregular designs of resolution IV or 
more containing clear two-factor interactions and presented necessary and sufficient conditions for 
the existence of such designs. They gave many designs in concise grid representations for A'^ = 48 
up to 192 and N being a multiple of 16. 

Stufken and Tang (2007) completely classified all two-level orthogonal arrays with t + 2 factors, 
strength t and any run size. The key tool they used is the theory of J-characteristics developed 
by Tang (2001). Cheng, Mee and Yee (2008) studied the construction of second-order saturated 
orthogonal arrays of strength three OA{N, 2"*, 3), which allows N — m — 1 two-factor interactions 
to be estimated besides m main effects. 

7.3 Nonregular designs constructed via quaternary codes 

The construction of large regular designs is known to be very difficult (Xu 2007) . The problem is 

even harder for nonregular designs. The construction via quaternary codes is relatively straight- 
forward and can generate good large nonregular designs. 

A quaternary code is a linear subspace over Z4 = {0,1,2,3} (mod 4), the ring of integers 
modulus 4. A surprising breakthrough in coding theory is that many famous nonlinear codes such 
as the Nordstrom and Robinson code can be constructed via quaternary codes (Hammons et al. 
1994). A key device is the so-called Gray map: 

0:0^ (0,0), 1^(0,1), 2 ^(1,1), 3 ^(1,0), 

which maps each symbol in Z4 to a pair of symbols in Z2. Let G be a A; x n matrix and let C 
consist of all possible linear combinations of the row vectors of G over Z4. Applying the Gray map 
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to C, one obtains a 4^^ x 2n matrix or a two-level design, denoted by D. Although C is linear over 
Z4, D may or may not be linear over Z2. 

Xu and Wong (2007) described a systematic procedure for constructing nonregular designs 
from quaternary codes. They first generated a A; x (4^^ — 2'^)/2 generator matrix G which has the 
following properties: (i) it does not have any column containing entries and 2 only and (ii) none 
of its column is a multiple of another column over Z4. Xu and Wong (2007) showed that the binary 
image D generated by G is a 4*^ x (4*^ — 2^) design with resolution 3.5 whereas regular designs of 
the same size have resolution 3. To obtain designs with less than 4*^ — 2^ columns, they developed 
a sequential algorithm, similar to those by Chen, Sun and Wu (1993) and Xu (2005b). They also 
presented a collection of nonregular designs with 32, 64, 128 and 256 runs and up to 64 factors, 
many of which are better than regular designs of the same size in terms of resolution, aberration 
and projectivity. 

Phoa and Xu (2008) further investigated the properties of quarter-fraction designs which can be 
defined by a generator matrix that consists of an identity matrix plus an extra column. They showed 
that the resolution, wordlength and projectivity can be calculated in terms of the frequencies of 
the numbers 1, 2 and 3 that appear in the extra column. These results enabled them to construct 
optimal quarter-fraction designs via quaternary codes under the max;imum resolution, minimum 
aberration and maximum projectivity criteria. These designs are often better than regular designs 
of the same size in terms of the design criterion. The generalized minimum aberration designs 
constructed via quaternary codes have the same aberration as the minimum aberration regular 
designs, and frequently with larger resolution and projectivity. A maximum projectivity design 
is often different from a minimum aberration or maximum resolution design but can have much 
larger projectivity than a minimum aberration regular design. They further showed that some of 
these designs have generalized minimum aberration and maximum projectivity among all possible 
designs. 

There are two obvious advantages of using quaternary codes to construct nonregular designs: 
(1) relatively straightforward construction and (2) simple design representation. Since the designs 
are constructed via linear codes over Z4, one can use column indexes to describe these designs. 
The linear structure of a quaternary code also facilitates the derivation and analytical study of 
properties of nonregular designs. 
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8 Concluding Remarks and Future Directions 



We have discussed recent developments in nonregular fractional factorial designs in the preceding 
sections. In a nutshell, when we compare regular designs with nonregular designs, nonregular 
designs have the following advantages: 

1. require smaller run size 

2. are more flexible in accommodating a variate number of factor levels 

3. have better geometrical or hidden projection properties 

4. have higher generalized resolution and projectivity 

5. have less generalized aberration 

6. lessen the contamination of nonnegligible two-factor interactions on the estimation of the 
main effects 

Some of the disadvantages of nonregular designs are that they are more complicated to analyze 
and some estimates of factorial effects may have larger variance than others. 

This review docs not include the developments in supersaturated designs, which arc factorial 
designs whose run sizes are not enough for estimating all the main effects. The research on su- 
persaturated designs has been very active since the influential work of Lin (1993) and Wu (1993). 
Broadly speaking, supersaturated designs are nonregular designs and optimality criteria such as 
generalized resolution and generalized minimum aberration can also be applied directly. As men- 
tioned earlier, Xu (2003) developed a unified theory for nonregular and supersaturated designs using 
the concept of minimum moment aberration. Xu and Wu (2005) obtained more theoretical results 
under the generalized minimum aberration criterion for multi-level and mixed-level supersaturated 
designs. Gilmour (2006) reviewed the recent development of two-level supersaturated designs for 
factor screening. 

Finally we highlight some future directions of research for nonregular designs and comment 
briefly why we feel they are useful: 

1. applications of nonregular designs 

2. analysis of nonregular designs 

3. construction of good nonregular designs with large run sizes 
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4. optimality results with respect to the generalized resolution. 

Despite significant developments in recent years and the advantages of using nonregular designs, 
they are still widely used for screening main effects only in practice and applications are largely 
limited to industry. We hope that by documenting recent advances in nonregular designs, our work 
may stimulate greater research interest in nonregular designs. We feel that there are opportunities 
that nonregular designs can be effectively applied to other fields to reduce experimental cost and 
gain improvement in statistical efficiency. 

The analysis of nonregular designs requires more attention. Although one can use any general 
purpose variable selection procedures, it is desirable to have user-friendly packages that incorporate 
the special features of nonregular designs in the analysis. More analysis strategies and comparisons 
are needed to further understand and utilize the complex aliasing structure of nonregular designs. 
Different procedures do not always lead to unequivocal conclusions. When that happens, extra 
runs are needed to resolve the ambiguity. See Meyer, Steinberg and Box (1996) and Box, Hunter 
and Hunter (2005, Chapter 7) for methods of constructing follow-up designs. 

There are plenty of catalogs of optimal nonregular designs with small run sizes (< 32). With 
the popularity of computer experiments, more and more large nonregular factorial designs will be 
used in practice. Mee (2004) illustrated how nonregular designs can be used to reduce run sizes 
significantly in applications requiring estimation of the main effects and two-factor interactions 
for a large number of factors. He suggested the use of a 2,096-run nonregular design with 47 
factors as an alternative to a 4,096-run regular designs in a ballistic missile simulation application. 
The quaternary code construction method is very promising in this regard and is able to produce 
large nonregular designs with good properties. Indeed, this 2,096-run nonregular design can be 
conveniently constructed via quaternary codes. 

Several optimality results and theories have been obtained for the minimum G2-aberration and 
the generalized minimum aberration. However, at the present time, there is very limited results 
on the generalized resolution for nonoregular designs. This is useful because it is always helpful to 
know whether a design is close to the optimal design or not, without knowing the optimal design. 
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